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(57) ABSTRACT 

The scanned document is analyzed by connected component 
extraction and by calculating bounding boxes for each 
connected component. The connected components repre- 
senting text, halftone images and characters overlapping the 
user-drawn circled region are eliminated to define candidate 
objects. The contour of candidate objects are generated and 
feature points are calculated including junctions of circles 
with other objects. Then, the user-drawn circled region is 
reconstructed by defining the contour segments belonging to 
the same circle. The userKlrawn circle is recognized through 
a series of tests based on the size as well as the area the 
contour covers. The circle is further defined by removing or 
disregarding open arcs. After the user-drawn circle is fiilly 
identified, the image inside the bounding box of the contour 
is extracted and optical recognition may be optionally per- 
formed on the extracted image. 

44 Claims, 9 Drawing Sheets 
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Request for Statutory Invention 

T^ppictra^jpn f<;TR) 




Statutory itivention registrati on^ 

(a) No^vithstanding any oihei' provision ot this title, the Commissioner is authorized to publish a statutory 
invention registration containing the specification and drawings of a regularly filed application for a patent without 
examination if the applicant 

(1 ) meets the requirements of section 1 12 of this title; 

(2) has complied with the requirements for printing, as set forth in regulations of the Commissioner; 

(3) waives the right to receive a patent on the invention within such period as may be prescribed by the 
Commissioner; and 

(4) pays application, publication, and other processing fees eslabl 

If an interference is dedaredwith respect to such an application, 
published unless the issue ofpriority of invention is finally dctcnnini 




(b) The waiver under subsection CaK3) of this section y an applicant shall take ciTccl 
statutory invention registration. 

(c) A statutory invention registration published pursi^t to this section shall have all of th Jbttribuies specified for 
patents in this title exceptthosc specified in sectio/ 183 and sections 271 through 28yf this title. A statutory 
invention registration shall not have any of the attiftuiiU»ttifiidi£iMttM&iUUttiAt^*r provision of law other 
than this title. A statutory invention registration published pursuant to this section shall give appropriate notice to 
the public, pursuant to regulations which the Commissioner shall issue, of the preceding provisions of this 
subsection. Tie invention with respect to which ostatutory invention certificate is published is not a patented 
invention for purposes of section 292 of this title. 

(d) The Secretary ofCommcrce shall report to the Congiem annualtyon the use of statutory invention registrations. 
Such report shall include an assessment of the degree to which agencies of the federal government are making use 
of the suiutory invention registration system, the degree to which it aids the management of federally developed 
technology, and an assessment of the cost savings to the Federal Government of the uses of such procedures. 

37 CFR i.293. Statutory invention registration. 

(a) An applicant for an original patent may request, at anytime during the pendency of applicant's pending 
complete application, that the specification and drawings be published as a statutory invention registration. Any 
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Figure 9A Figure 9B 
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USER DRAWN CIRCLED REGION key geometric features of the traced shape and then gener- 

EXTRACTION FROM SCANNED ales a substitute circle using the identified points as a guide. 

DOCUMENTS The substitute circle, having been computer generated, 

forms a fully closed circle structure that may be more 

BACKGROUND AND SUMMARY OF THE 5 reliably relied upon for extracting text contained within the 

INVENTION circled region. 

The present invention relates generally to the electronic 1° instances where the user does not draw a fuUy closed 

documents and imaging. More particularly, the invention circle, the system can employ one of a variety of endpoint 

relates to a system and method for identifying a user-circled proximity algorithms to ascertain whether the users intent 

region on a scanned document or bit mapped image, to allow ^° was to draw a fuUy enclosed circle, or not. 

information within the encircled region to be further ana- For a more complete understanding of the invention, its 

lyzed as by optical character recognition. objects and advantages, refer to the following specification 

With the rapid development of computer technologies, and to the accompanying drawings, 
more and more docimients have been archived in electronic 

form.Theelectronicformraisesanentirelynewsetofissues BRIEF DESCRIPTION OF THE DRAWINGS 

concerning information retrieval. ^ ^ ^^^^^^^ ^ document page, showing a 

Unlike alphanumeric text files, which can be keyword iiser<ircled region- 

searched easUy files coDtaining bitmapped images or FIG. 2 is an example of an electronic document displayed 

scanned mformation cannot be so readity searched This is ^ by use of a stylus; 

because bitmapped images and scanned documents repre- ^ ^ 

sent printed words as graphical pictures made up of tiny ^ illustrates a document image database system in 

black, white or colored dots that are not directly recogniz- accordance with the mvenuon; 

able by a computer as characters or letters of the alphabet. FIG. 4 is a flowchart illustrating the preferred steps for 

Fortunately, in many cases these graphical pictures of char- ^5 extracting circled regions from scanned or bitmapped docu- 

acters and letters can be converted into computer- ments; 

recognizable text characters by employing optical character FIG. 5 is an example of overlapping objects; 

recognition (OCR) software. Even with fast computers, the pjc 5 ^5 ^n example of the overlapping objects of FIG, 

optical character recognition conversion process is a slow 5^ showing contours and feature points; 

one. In many cases it is simply not practical to convert entire „ ™^ _. -.i * *• c- • j * * 1 

,/ f J J ^ . 1 L ^0 FIG. 7 IS an illustration of mcommg and outgomg angles; 

bitmapped image for scanned documents using optical char- * & & & » 

acter recognition software. 8 is an example of reconnecting according to the 

In cases where it is not practical to employ optical Pi^^rred smoothness algorithm; 
character recognition conversion, document image data- FIGS, 9A-9D illustrate four (4) different cases for con- 
bases may be indexed by the user manually assigning a ^5 '^^^^S P) segments; 

character-based text name or label to each bitmapped image FIG. 10 illustrates the reconnecting of feature points using 

or scanned image. The computer then associates the text the example of FIGS. 5 and 6; 

name or label with the image data, so that the image data can picS. IIA-IIC illustrate three (3) alternate embodiments 

be later retrieved by searching upon the text name or label. of a closed loop criterion function; 

Manual entry of appropriate text names and labels is labor 40 pi^s 12A-12G illustrate examples of accepted and 

intensive and often requires additional computer hardware, rejected objects; 

including keyboard or numeric keypad connected to the ^ia i\t% -j *. 1 r 1 • 
J . • • , rr 1 . *r u I 1 . .u • -1 FIGS. 13A-13B provide two (2) examples for closmg a 
document miaging stotion. Taking a fresh look a the mdex- ^ user-drawn circle for display 
mg problem, some have suggested embeddmg technology ^ ^ ^ ^ 
within the document imaging system that would recognize 45 DESCRIPTION OF THE PREFERRED 
user-circled regions. Such regions, if identified, could serve EMBODIMENT 
to localize, and thus minimize, the optical character recog- 
nition process. The user would circle the graphical image of The system and method for extracting user-drawn circled 
a desired keyword, and the system would then perform regions can be applied in a wide variety of different appli- 
optical character recognition on the image data within the 50 cations. FIG. 1 illustrates an example of a document page 
circle, converting it into a label that would then be associ- with two (2) user-drawn circled regions at 20 and 22. Note 
ated with the image data file. that circle 20 is partially closed and circle 22 overlaps with 
Unfortunately, while the idea has merit, it has proven an adjacent rectangular annotation. The document can be 
quite difBcult to accurately and repeatedly detect the user- suitably digitized using a variety of different devices, includ- 
circled region, particularly given the wide variation in 55 ing scanners, fax machines, digital photocopy machines, and 
handwriting quality from one user to the next. The problem the like. Once digitized, the document is represented as a 
is further compounded by the fact that some user-drawn bitmapped image, or scanned image. As will be more fully 
circles may overlap or intersect with other circles (whether explained, the extraction system and method locates the 
user-drawn or not); and because some user-drawn "circles" user-drawn circled regions and extracts the encircled region 
may not be fully closed. The latter problem can arise where 60 content. The content may then be further processed as with 
the user simply does not fully finish drawing a circle, or optical character recognition software. The user-drawn 
where the user-drawn circle is broken by image dropout. circles can be applied using any suitable drawing instru- 
Image dropout can occur, for example, when a light or thin ment. 

circle is drawn and then processed by low resolution scan- Alternatively, the document can be an electronic 

ning equipment. 65 document, displayed upon the screen of a computer or 

llie present invention addresses these problems by a document imaging display, for example. FIG. 2 illustrates an 

system and method that "traces" a circle candidate, identifies electronic document displayed upon the screen of a laptop 
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computer 26 and in which the user-drawn circles are formed 
using a stylus 28 to draw on digitizing pad 30. 

The embodiment of FIG. 2 illustrates that the document 
does not necessarily need to originate as a paper document. 
Rather, the document can be generated by software that 
creates a bitmapped image, such as any of the popular 
paintbrush programs or photo manipulation programs. 
Alternatively, the document can be input using a digital 
camera or video camera. To illustrate, FIG. 2 shows a digital 
camera 32 connected to computer 26. This camera can be 
used to take a picture of the document of interest. Thereafter, 
the user draws circled regions using stylus 28. 

To further illustrate some of the possible uses for the 
invention, refer to FIG. 3, which shows an exemplary 
electronic document imaging system. The system includes a 
document imaging server 40 that has an associated storage 
medium such as a disk array 42. The server is coupled 
through suitable network communication link 44 to a variety 
of document input devices and viewing devices. For 
example, a digital copy machine 46 or flatbed scanner 48 can 
be used to convert paper documents into electronic fonn for 
storage by the server 40. Once digitized and stored, the 
document images may be viewed on any suitable worksta- 
tions such as computer workstation 50. 

Documents are stored in the document image database as 
a series of digital image data records to which an associated 
index label is attached, as illustrated diagrammatically at 52. 
The user-drawn circled region extraction system of the 
invention uses the extracted region to generate the index 
label by which the digital image data may be later retrieved. 
The region extraction system software may be hosted by any 
suitable processor in the system. Such processors include, 
for example, the server 40, the workstation 50, and even the 
processors associated with the copy machine 46 or flatbed 
scanner 48. 

The presently preferred embodiment of the circled region 
extraction process is illustrated in FIG. 4. 

The presently preferred contoiu' analysis technique is 
illustrated in FIG. 4. The technique is based on identifying 
the contour of a candidate object. The candidate object may 
include user-drawn circles, machine printed frames or other 
non-text materials on the page. By analyzing the contour 
using a feature extraction module, the algorithm extracts the 
user-drawn circle. The contour is represented by a sequence 
of points with different curvature values. This makes it 
possible to distinguish a user-drawn circle from a machine- 
printed frame through a post processing analytical tech- 
nique. Referring to FIG. 4, to speed up the algorithm further, 
an optional pre-processing step may be performed at 250. 
The first step (step 252) is to find the connected components 
within a given page. In this step the image is scanned 
line-by-line and each pixel is labeled if it is connected with 
its neighboring pixels. After labeling the pixels for 
connectivity, a bounding box is calculated for each con- 
nected component. These bounding boxes are used to extract 
the candidate area in the image. 

Next (at step 254) connected components representing 
text are eliminated. This is accomplished by analyzing the 
size of the bounding box and eliminating those connected 
components that have bounding boxes below a predeter- 
mined size. Eliminating connected components that are not 
likely user-drawn circle candidates helps speed up subse- 
quent processing. However, if suflScient processing power is 
available, step 254 can be eliminated. 

Next, halftone images are eliminated at step 256. Halftone 
images compose large connected components. The algo- 
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rithm detects halftone images by assessing the black-to- 
white pixel ratio within the bounding box associated with 
the cormected component in question. Halftone images tend 
to have more black pixels than areas containing text. Again, 
5 if suf&cient processing power is available, it may not be 
necessary to eliminate half tone images at step 256. The 
preferred embodiment eliminates half tone images as user- 
drawn circle candidates, thus significantly improving pro- 
cessing speed. 

10 The contour analysis technique may generate more fea- 
ture points than are actually required when there are extra- 
neous characters touching the contour in question. This can 
occur, for example, when the user draws a circle that 
happens to intersect with other characters on the page. 

The optional pre-processing step eliminates these touch- 
ing characters by performing a morphological operation to 
simplify the shape of the contour in the region where the 
user-drawn circle and extraneous character intersect. 

Next, the user-drawn circle is identified by examining 
different candidate areas on the page. The first step in this 
procedure, depicted at 258, involves generating the contour 
of the candidate object. This is performed by tracing the 
outline of the object. 

By way of illustration, refer to FIG. 5, which shows an 
example of two (2) candidate areas. Note that these areas 
happen to overlap, which makes the user-drawn circle 
detection problem more difScult. Also note that shape 60 has 
a gap at 61 where the circle was either not fiilly formed or 
contains a dropout or some other defect. The algorithm must 
determine which of the contours is the user-drawn circle. In 
FIG. 5 there are four (4) possibilities: the eUiptical shape 60, 
the rectangular shape 62, the overlapping intersection 64 of 
shapes 60 and 62, and the combined outer boundary shape 

^5 66 of shapes 60 and 62 combined. 

The contour is represented in computer memory as an 
ordered set of points (coordinates of the boundary pixels). 
The tracing scheme first scans for the starting point (a border 
pixel that is not previously traced). Then the trace starts in 

^ either a clockwise or cotmter-clockwise direction along the 
convex outline of the object. When the trace goes back to the 
starting point, or to a point where no more black pixels can 
be found around the current one, the trace stops and the 
scanning process to find the next starting point continues. 

45 FIG. 6 shows an example of the trace contours that would 
result when the tracing step 258 is performed on the over- 
lapping objects of FIG. 5. Note that the contour tracing 
appears as closely spaced double lines that follow the inside 
and outside perimeters of the lines that define the objects. 

50 Next the contours obtained during step 258 are analyzed 
by calculating feature points associated with each contour 
(step 260). Contours obtained from step 258 can be closed 
curves or broken curves, due to noise in the image. Feature 
points are defined as high curvature points, including junc- 

55 tions of circles with other objects. Curvature can be calcu- 
lated using re-sampling techniques, however this may not be 
reliable when noise is generated during the digitization 
process. By observation, the feature points can be detected 
approximately at either the local maxima or local minima on 

60 the x and y direction, even though not all maxima and 
minima are feature points. In the preferred implementation 
the starting point and ending point of each contour are 
treated as feature points. FIG. 6 shows the applicable feature 
points at 70-86. 

65 After feature points have been identified, the circled 
region is reconstructed using the feature points. This is 
illustrated at step 262. In essence, each contour generated at 
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Step 258 is broken into segments at the feature points. Tliese the size of the contour as well as the area the contour covers, 

segments are examined and reconnected such that segments If the length of the contour exceeds a predetermined thresh- 

belonging to different objects are separated and those old and the bounding box of the contour covers a predeter- 

belonging to the same object are connected. mined area, the algorithm considers the contour to be a 

The main criterion for reconnecting the contour segments 5 user-drawn circle, 

is to check the smoothness when making the transition However, in order to discriminate between user-drawn 

between neighboring segments. For each contour segment, a circles and machine-printed frames, such as rectangular 

small area around its starting and ending points is examined. frames, machine-printed circles or tables, the smoothness of 

The points on the two ends are fitted into lines so that the connected contour is examined. One way to measure 

slope (angles coming and going out of the segment) can be lO smoothness is to calculate the average curvature along the 

estimated. These estimated angles are tised to assess whether connected contour. If smoothness exceeds a predetermined 

two line segments are approaching one another such that threshold, the contour is considered to be machine-printed, 

they should be connected as belonging to the same contour Smoothness scan be expressed as: 

or are crossing one another such that they belong to unre- _ 

latedcontou^ 15 Equauon 1 

FIG. 7 shows how each contour segment is assessed. The 

starting and ending points are examined by fitting the if c^T^^^^^ machine generated 
endpoints into lines, so that the slope may be assessed. In 

FIG. 7, angle 1 is the incoming angle and angle 2 is the if c<r^^^^ user drawn 

outgoing angle. The direction of the contour segment in FIG. r^.^^ threshold 
7 is firom a to b. The incoming angle is estimated by fitting 

the first few points of the segment into a line and calculating ^ = V aO> ao is the curvature value along the curve 

its slope. The outgoing angle is estimated in a similar way V 
by fitting the last points of the segment into a line. The 
angles are defined such that they are in a range of +or-180'*, 

For each segment contour, the algorithm examines aU the , because the algorithm recotistructs circled regions from 

starting and ending points that are located in its neighbor- calculated feature pomts, it is able to identify «ser-drawn 

hood areas. If there is only one neighbor, the algorithm does •'^f" they contam small gaps or breaks. If desired, 

not need to check the smoothness of the changing of the "ser^rawn circle candidates can be further evaluated to 

angles. Hie algorithm simply connects the points together. If ^ determme if any gaps are sufficiently large to waixant 

there are multiple neighbors, then the algorithm chooses the '«J^«='?°° ^ user-drawn circles. The analysis (depicted at 

one neighbor that makes the smoothest transition. This is 266) mvolves assessing the distant between the starting 

illustrated in FIG. 8. In FIG. 8 starting and ending points a P~°' "="''"•6 P°f°J " ""stance can be 

and b would thus be connected. assased in a variety of different ways. 

« . , r . « 35 One techmque for assessmg distance is to determine 

HGS. 9A-9D Illustrate several cases for connectmg^e ^ ^ predetermined radial 

two (2) segments when the smoothness crUenon ^ met. We ^^^^^ ^^^^ ^ ^ ^^^^ ,^ 

refer to the left segment as segment 1 and ihc right segment ^ ^ ^^^^ ^j^^^^ ^^^^^^ ,^ 

to segment 2 m the followmg descnpUon. Tlie dots appear- ^^^^ ^ ^^^^^ predetermined size around 

ing on the segments represent the starting points, Tlie ^ one end point and to determine whether the other end point 

property of smoothness may best be defined as foUows: ^ PIO ^^^^ ^ 

(a) Smoothness is defined as the difference of the outgoing ^ ^^^^ distance function. A third technique is to define 
angle of segment 1 and the mcommg angle of segment ^ ^^^^ bounding box of predetermined size around one end 
2. If the smoothness condition is satisfied, append ^^^^ ^^^^ jo rotate the bounding box around that end 
segment 2 to segment 1. 45 j^Jq, determine if at any rotational orientation the second 

(b) Smoothness is defined as the difference of the outgo- end point falls within the bounding box. This will occur, if 
ing angle of segment 1 and the reverse of the outgoing at ^hen one comer of the bounding box lies on a Une 
angle of segment 2. If the smoothness condition is between the two end points. See ¥IG, IIC. We call this the 
satisfied, reverse segment 2 and then append to seg- Manhattan distance function. 

0^®°^ 1* 50 If the contour fails to pass any of the above tests, then it 

(c) Smoothness is defined as the difference of the reverse is considered to be an open arc, as opposed to a user-drawn 
of the incoming angle of segment 1 and the incoming circle. Illustrations of accepted circles are shown in FIGS, 
angle of segment 2. If the smoothness condition is 12A-12C Rejected objects are shown in FIGS. 12D-12G. 
satisfied, reverse segment 1, then append segment 2 to Once the user-drawn circle is identified as described above, 
segment 1. 55 the bounding box around its contour is used to delimit the 

(d) Smoothness is defined as the difference of the incom- region that will be extracted for subsequent optical character 
ing angle of segment 1 and the outgoing angle of recognition analysis. This is illustrated at step 268. The 
segment 2. If the smoothness condition is satisfied, image inside the bounding box of the contour is extracted 
append segment 1 to segment 2. and optical character recognition may be optionally per- 

Based on these criteria, the algorithm reconnects the 60 formed on the extracted image, 

contour segments that are split at the feature points. FIG. 10 As illustrated in FIG. 6, a user-drawn circle has two (2) 

shows the results after reconnection, in which process the edges that thus causes two (2) sets of contours to be 

segmented contours are formed into two (2) different generated, one inside the other. The algorithm uses the larger 

objects, elliptical object 90 and rectangular object 92. of the two (2) bounding boxes for region extraction. 

Referring once again to FIG. 4, using the circles reoon- 65 In a document imaging system, the extracted circled 

structed at step 262, the user-drawn circles are identified at region can be displayed in a variety of formats. One way is 

step 264 through a series of tests. The first test is based on to output the circled regions as sub-images. Another is to 
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output the entire page, with the user-circled regions high- 
lighted in color. A further option is to have the inside of the 
circle highlighted. To do this, the gaps on the contour must 
be closed. Closure may be handled in several ways. Two (2) 
are illustrated in FIGS. 13A and 13B. In 13A, closure is 5 
effected by connecting the end points directly. In FIG. 13B 
closure is effected by having one end point connect with the 
point on the curve that has the minimum distance between 
the end point and the curve. 

In addition to extracting the user-drawn region for display, 10 
the system can also process the extracted region using 
optical character recognition techniques to construct index 
labels for document storage and retrieval purposes. 

While the invention has been described in its presently 
preferred embodiments, it will be understood that certain 15 
modifications can be made to these embodiments without 
departing from the spirit of the invention as set forth in the 
appended claims. 

What is claimed is: 

1. A computer-implemented method of extracting a user- 20 
drawn circled region within image data, the image data 
having at least one candidate object, comprising the steps of: 

generating a contour representative of a candidate object; 

calculating feature points for said contour of said candi- 
date object; 25 

reconstructing said user-circled region based on said 
feature points of said contour; and 

defining a transition smoothness metric and using said 
metric in constructing said user-circled region, wherein 
said transition smoothness metric: 

(a) identifies incoming and outgoing segments; 

(b) determines an angle associated with said incoming 
and outgoing segments; and 

(c) associates an incoming segment with an outgoing 
segment if their respective associated angles are 
within a predetermined relationship to one another. 

2. The method of claim 1 wherein said feature points are 
defined as high curvature points which are detected approxi- 
mately at either the local maxima or local minima on the x 
and y directions. ^ 

3. The method of claim 1 wherein said step of recon- 
structing said user-circled region further comprises the steps 
of: 

dividing said contour of said candidate object into seg- 
ments at said feature points; 

examining the transition smoothness between segments; 

separating said segments belonging to different candidate 
objects; and 

connecting said segments belonging to the same candidate 
objects. 

4. The method of claim 1 further comprising the step of: 
when reconstructing the circles using the feature points, 

analyzing if an openloop occurs in said first candidate 
object using a near closed loop criterion function. 55 

5. The method of claim 1 for discriminating said user- 
drawn circled region from a machine printed frame further 
comprising: 

comparing the smoothness of said contour of said candi- 
date object with a predetermined value representative 60 
of a smoothness of said machine printed frame. 

6. The method of claim 1 wherein said step of identifying 
said user-drawn circled region further comprises the steps 
of: 

comparing the smoothness of said contour of said candi- 65 
date object with a predetermined value representative 
of a smoothness of a machine printed frame; and 
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analyzing if an openloop occurs in said candidate object 
using a near closed loop criterion function. 

7. The method of claim 4 wherein said step of analyzing 
if an openloop occurs further comprises the steps of: 

assessing a distance between a starting point of said first 
candidate object and an ending point of said first 
candidate object; 

performing a circular distance function; 

performing a square distance function; or 

performing a Manhattan distance function wherein said 
first candidate object is defined as said user-drawn 
circled region only if said first candidate object passes 
one of said three functions. 

8. The method of claim 7 wherein said circular distance 
function determines whether said starting point of said first 
candidate object is within a predetermined radial distance 
from said ending point of said first candidate object. 

9. The method of claim 7 wherein said square distance 
function determines whether said ending point of said first 
candidate object is within a square bounding box of prede- 
termined size around said ending point. 

10. The method of claim 7 wherein said Manhattan 
distance function rotates said square bounding box of pre- 
determined size around said ending point of said first 
candidate object in order to determine whether said starting 
point falls within said bounding box at any rotational 
orientation. 

11. The method of claim 1 wherein said step of generating 
said contour representative of said candidate object further 
comprising the steps of: 

identifying connect components within the image data; 
calculating a bounding box for each of said connected 
components; 

eliminating text fi^om each of said bounding boxes by 
analyzing the size of each bounding box; 

eliminating halftone images by assessing the black-to- 
white pixel ratio within said bounding boxes remaining 
after said elimination of text; 

extracting at least one candidate object from said bound- 
ing boxes remaining after said elimination of halftone 
image prior to the step of generating said contours; and 

tracing outline of said candidate objects to generate 
contours. 

12. The method of claim 5 further comprising the step of 
eliminating characters touching said candidate object by 
performing a morphological operation to simplify a shape of 
said candidate object where said candidate objects and said 
character intersect. 

13. The method of claim 1 further comprising the step of 
extracting an image inside said user-drawn circled region. 

14. A computer-implemented method of extracting a 
user-drawn circled region within image data, comprising: 

extracting a candidate object from said image data; 

applying an openloop criterion function to said candidate 
object to identify and assess the relationship between a 
starting point and an ending point associated with said 
candidate object; 

identifying said candidate object as a user drawn circle if 
said starting point and said ending point bear at least 
one predetermined relationship to each other. 

15. The method of claim 14 wherein said at least one 
predetermined relationship employs a circular distance func- 
tion. 

16. The method of claim 14 wherein said at least one 
predetermined relationship employs a square distance func- 
tion. 
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17. The method of claim 14 wherein said at least one 
predetermined relationship employs a Manhattan distance 
function. 

18. A computer- implemented method of extracting a 
user-drawn circled region within image data, the image data 
having at least one candidate object, comprising the steps of: 

generating a contour representative of a candidate object; 

calculating feature points for said contour of said candi- 
date object; and 

reconstructing said user-circled region based on said 
feature points of said contour, wherein said reconstruct- 
ing includes analyzing if an openioop occurs in said 
first candidate object using a near closed loop criterion 
function, and said analyzing includes: 

(a) assessing a distance between a starting point of said 
first candidate object and an ending point of said first 
candidate object; 

(b) performing a circular distance function; 

(c) performing a square distance function; or 

(d) performing a Manhattan distance function wherein 
said first candidate object is defined as said user- 
drawn circled region only if said first candidate 
object passes one of said three functions. 

19. The method of claim 18 wherein said feature points 
are defined as high curvature points which are detected 
approximately at either the local maxima or local minima on 
the X and y directions. 

20. The method of claim 18 wherein said step of recon- 
structing said user-circled region further comprises the steps 
of: 

dividing said contour of said candidate object into seg- 
ments at said feature points; 

examining the transition smoothness between segments; 

separating said segments belonging to different candidate 
objects; and 

connecting said segments belonging to the same candidate 
objects. 

21. The method of claim 18 further comprising: 
defining a transition smoothness metric and using said 

metric in constructing said user-circled region. 

22. The method of claim 18 wherein said transition 
smoothness metric: 

identifies incoming and outgoing segments; 

determines an angle associated with said incoming and 
outgoing segments; and 

associates an incoming segment with an outgoing seg- 
ment if their respective associated angles are within a 
predetermined relationship to one another. 

23. The method of claim 18 for discriminating said 
use-drawn circled region from a machine printed frame 
further comprising: 

comparing the smoothness of said contour of said candi- 
date object with a predetermined value representative 
of a smoothness of said machine printed frame. 

24. The method of claim 18 wherein said step of identi- 
fying said user-drawn circled region further comprises the 
steps of: 

comparing the smoothness of said contour of said candi- 
date object with a predetermined value representative 
of a smoothness of a machine printed frame; and 

analyzing if an openioop occurs in said candidate object 
using a near closed loop criterion function. 

25. The method of claim 18 wherein said circular distance 
fimction determines whether said starting point of said first 
candidate object is within a predetermined radial distance 
from said ending point of said first candidate object. 
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26. The method of claim 18 wherein said square distance 
function determines whether said ending point of said first 
candidate object is within a square bounding box of prede- 
termined size around said ending point. 

27. The method of claim 18 wherein said Manhattan 
distance function rotates said square bounding box of pre- 
determined size around said ending point of said first 
candidate object in order to determine whether said starting 
point falls within said bounding box at any rotational 
orientation. 

28. The method of claim 18 wherein said step of gener- 
ating said contour representative of said candidate object 
further comprising the steps of: 

identifying connect components within the image data; 
calculating a bounding box for each of said connected 
components; 

eliminating text from each of said bounding boxes by 
analyzing the size of each boimding box; 

eliminating halftone images by assessing the black-to- 
white pixel ratio within said bounding boxes remaining 
after said elimination of text; 

extracting at least one candidate object from said bound- 
ing boxes remaining after said elimination of halftone 
image prior to the step of generating said contours; and 

tracing outline of said candidate objects to generate 
contours. 

29. The method of claim 23 further comprising the step of 
eliminating characters touching said candidate object by 
performing a morphological operation to simplify a shape of 
said candidate object where said candidate objects and said 
character intersect. 

30. The method of claim 18 further comprising the step of 
extracting an image inside said user-drawn circled region. 

31. A computer-implemented method of extracting a 
user-drawn circled region within image data, the image data 
having at least one candidate object, comprising the steps of: 

generating a contour representative of a candidate, object; 

calculating feature points for said contour of said candi- 
date object; and 

reconstructing said user-circled region based on said 
feature points of said contour, 

wherein said step of generating said contour representa- 
tive of said candidate object comprises the steps of: 

(a) identifying connect components within the image 
data; 

(b) calculating a bounding box for each of said con- 
nected components; 

(c) eliminating text from each of said bounding boxes 
by analyzing the size of each bounding box; 

(d) eliminating halftone images by assessing the black- 
to-white pixel ratio within said bounding boxes 
remaining after said elimination of text; 

(e) extracting at least one candidate object from said 
bounding boxes remaining after said elimination of 
halftone image prior to the step of generating said 
contours; and 

(0 tracing outline of said candidate objects to generate 
contours. 

32. The method of claim 31 wherein said feature points 
are defined as high curvature points which are detected 
approximately at either the local maxima or local minima on 
the x and y directions. 

33. The method of claim 31 wherein said step of recon- 
stmcting said user-circled region further comprises the steps 
of: 
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dividing said contour of said candidate object into seg- analyzing if an openloop occurs in said candidate object 

ments at said feature points; using a near closed loop criterion function, 

examining the transition smoothness between segments; J°^t*^od of claim 34 wherein said step of analyzing 

. _ . , . j.fl. J.J if an openloop occiirs ^rther comprises the steps of: 

separating said segments belonging to different candidate . ... . , ^ ^. ■ . r j c * 

h' L • d ^ assessmg a distance between a startmg pomt of said first 

objects; and candidate object and an ending point of said first 

connecting said segments belonging to the same candidate candidate object; 

objects. performing a circular distance function; 

34. The method of claim 31 further comprising the step of: performing a square distance function; or 

when reconstructing the circles using the feature points, jq performing a Manhattan distance function wherein said 

analyzing if an openloop occurs in said first candidate fijst candidate object is defined as said user-drawn 

object using a near closed loop criterion function. circled region only if said first candidate object passes 

35. The method of claim 31 further comprising: one of said three fiinctions. 

defining a transition smoothness metric and using said 40. The method of claim 39 wherein said circular distance 

metric in constmcting said user-circled region. is function determines whether said starting point of said first 

36. The method of claim 35 wherein said transition candidate object is within a predetermined radial distance 
smoothness metric: &om said ending point of said first candidate object. 

identifies incoming and outgoing segments; ^l- Tb^ method of claim 39 wherein said square distance 

determines an angle associated with said incoming and ^"^^^^ determines whether said ending point of said first 

outgoing segments' and ^° candidate object is withm a square boundmg box of prede- 

. . .* termined size around said ending point, 

associates an mooming segment with an outgomg seg- The method of claim 39 wherein said Manhattan 

ment if their respecUve associated angles are within a ^^^^^^ ^^^^^^^ ^^^^^^^ ^^^^ bounding box of pre- 

predetermmed relauonship to one another. determined size around said ending point of said first 

37 TTie method of claim 31 for discnmmatmg said candidate object in order to determine whether said starting 

useKlrawn circled region from a machine pnnted frame • j bounding box at any rotational 

further comprising: orientation, 

comparing the smoothness of said contour of said candi- 43. The method of claim 37 further comprising the step of 

date object with a predetermined value representative eliminating characters touching said candidate object by 

of a smoothness of said machine printed firame. 30 performing a morphological operation to simplify a shape of 

38. The method of claim 31 wherein said step of identi- said candidate object where said candidate objects and said 

fying said user-drawn circled region further comprises the character intersect. 

steps of: 44, The method of claim 31 further comprising the step of 

comparing the smoothness of said contour of said candi- extracting an image inside said user-drawn circled region, 

date object with a predetermined value representative 35 

of a smoothness of a machine printed frame; and * « « « « 



04/25/2004, EAST Version: 1.4.1 



UNITED STATES PATENT AND TRADEMARK OFHCE 

CERTIFICATE OF CORRECTION 



PATENT NO. : 6.597,808 Bl Page 1 of 1 

DATED : July 22. 2003 

INVENTOR(S) : Jinhong Guo et al. 



It is certified that error appears in tiie above- identified patent and that said Letters Patent is 
hereby corrected as shown below: 



Column 9, 

Line 52, "use-drawn" should be - user-drawn - 
Column 10, 

Line 39, "candidate," should be - candidate - 
Column 11, 

Line 26, "use-drawn" should be -- user-drawn - 



Signed and Sealed this 
Twenty-fifth Day of November, 2003 

JAMES E. ROGAN 
Director of the United States Patent and Trademark Office 



04/25/2004, EAST Version: 1.4.1 



UNITED STATES PATENT AND TRADEMARK OFRCE 

CERTIFICATE OF CORRECTION 



PATENT NO. : 6,597,808 Bl Page 1 of 1 

DATED : July 22, 2003 

INVENTOR(S) : Jinhong Guo et al. 



It is certified that error appears in the above-identified patent and that said Letters Patent is 
hereby corrected as shown below: 



Title page. 

Item [73], Assignee, "Matsushita Electric Industrial Co,, Ltd., Osaka (JP)" should be 
-- Matsushita Electric Corporation of America, Pinceton, NJ (US) ~ 

Column 9. 

Line 52, "use-drawn" should be -- user-drawn 
Column 10, 

Line 39, "candidate," should be - candidate 
Column 11, 

Line 26, "use-drawn" should be - user-drawn -- 



Signed and Sealed this 
Thirteenth Day of April, 2004 

JON W. DUDAS 
Acting Director of the United States Patent and Trademark Office 



04/25/2004, EAST Version: 1.4.1 



